Skip to content

Intake wdl#53

Open
piotor87 wants to merge 10 commits into
masterfrom
intake-wdl
Open

Intake wdl#53
piotor87 wants to merge 10 commits into
masterfrom
intake-wdl

Conversation

@piotor87

@piotor87 piotor87 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor
  • removed a bunch of legacy folders from wdl since we have releases where one can look
  • edited intake to now process all columns (we will decide later what to keep anyways if needed)
  • change _rowid to ROWID since it will be already that downstream anyways
  • added intake.wdl/json to process the data and outputs parquet +tsv.gz file

@piotor87 piotor87 requested a review from vincent-octo June 30, 2026 12:00
vincent-octo
vincent-octo approved these changes Jul 2, 2026
View reviewed changes

@vincent-octo vincent-octo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good at this point. :)

I will add a command to run the whole thing end-to-end in order to keep the WDL small.

One thing we should discuss at some point: should we just keep _rowid_source instead of ROWID. The advantage of _rowid_source is that it's a direct lookup in the intermediate files generated.

vincent-octo
vincent-octo reviewed Jul 2, 2026
View reviewed changes
Comment thread docker/Dockerfile Outdated
@piotor87

piotor87 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

These are the changes

src/kanta/intake/tidyup.py — the actual OOM fix
  - The old code built one big lazy plan (join SEX + with_row_index + sanitize + sink_parquet) over all buckets concatenated together.
  maintain_order="left" on the join and the global counter in with_row_index both disable Polars' streaming engine, forcing full materialization in
  memory.
  - Now each bucket is joined, indexed (with a running ROWID offset carried across buckets), sanitized, and sunk to its own temp parquet file individually
  — bounded memory per bucket regardless of total dataset size.
  - Final buckets (and duplicate buckets) are concatenated into the output file via a new concatenate_parquet_files helper that streams row batches
  through pyarrow directly, since sink_parquet over a multi-file scan_parquet isn't reliably streaming either.
  - Docstring note updated to mention the join+sanitize stage also processes one bucket at a time now.
 
  docker/Dockerfile
  - Bumped Python 3.12 → 3.14.
  - Replaced manual uv venv + uv pip install with uv sync --python 3.14 --locked (via UV_PROJECT_ENVIRONMENT), so the venv is built straight from the
  lockfile.

vincent-octo
vincent-octo reviewed Jul 3, 2026
View reviewed changes
Comment thread src/kanta/intake/tidyup.py
args = init_cli()

output.check_safe_write(args.output_file)
output.check_safe_write(args.output_file.with_name(f"{args.output_file.stem}_duplicates.parquet"))

@vincent-octo vincent-octo Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line

output.check_safe_write(args.output_file.with_name(f"{args.output_file.stem}_duplicates.parquet"))

should also be added to

kanta_lab_preprocessing/src/kanta/intake/__main__.py

Lines 54 to 56 in 9bf7288

output.check_safe_write(output_file_assemble_stage)
output.check_safe_write(output_file_tidyup_stage)

@piotor87 piotor87 Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok i can add it

    output.check_safe_write(output_file_assemble_stage)
    output.check_safe_write(output_file_tidyup_stage)
    output.check_safe_write(
        output_file_tidyup_stage.with_name(f"{output_file_tidyup_stage.stem}_duplicates.parquet")
    )

@piotor87

piotor87 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

I've also split the tsv generation in the wdl because it made sense structurally. Otherwise I think we're good to go

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vincent-octo vincent-octo vincent-octo approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@piotor87 @vincent-octo